Predicting Scientific Grid Data Transfer Characteristics

Big data scientists routinely transfer massive amounts of data. By understanding and modelling different aspects of these data transfers, we can make using big data more efficient and user-friendly. In this paper, we first develop a set of data storage location prediction heuristics. These heuristics help big data scientists manage and discover locations to transfer their data from and to. We show, via analysis of historical Globus operations, that our approaches can predict the storage locations accessed by users with 78.2% and 95.5% accuracy for top-1 and top-3 recommendations, respectively. Predicting transfer bandwidth allows for more optimal selections of data replicas to download from and for more optimal scheduling and routing of data transfers. We show that existing bandwidth prediction techniques perform poorly on real-world data and develop heuristics that (performance statistics).